This notebook guides model developers through the process of automatically documenting and testing time series forecasting models. It shows you how to use the ValidMind Developer Framework to import and prepare data and before running a data validation test suite, followed by loading a pre-trained model and running a model validation test suite.
As part of the notebook, you will learn how to:
ValidMind’s platform enables organizations to identify, document, and manage model risks for all types of models, including AI/ML models, LLMs, and statistical models. As a model developer, you use the ValidMind Developer Framework to automate documentation and validation tests, and then use the ValidMind AI Risk Platform UI to collaborate on model documentation. Together, these products simplify model risk management, facilitate compliance with regulations and institutional standards, and enhance collaboration between yourself and model validators.
If this is your first time trying out ValidMind, we recommend going through the following resources first:
To access the ValidMind Platform UI, you’ll need an account.
Signing up is FREE — Create your account.
If you encounter errors due to missing modules in your Python environment, install the modules with pip install, and then re-run the notebook. For more help, refer to Installing Python Modules.
The client library provides Python support for the ValidMind Developer Framework. To install it:
%pip install -q validmindWARNING: You are using pip version 22.0.3; however, version 24.0 is available.
You should consider upgrading via the '/Users/andres/code/validmind-sdk/.venv/bin/python3 -m pip install --upgrade pip' command.Note: you may need to restart the kernel to use updated packages.ValidMind generates a unique code snippet for each registered model to connect with your developer environment. You initialize the client library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.
Get your code snippet:
In a browser, log into the Platform UI.
In the left sidebar, navigate to Model Inventory and click + Register new model.
Enter the model details, making sure to select Time Series Forecasting as the template and Credit Risk - Underwriting - Loan as the use case, and click Continue. (Need more help?)
Go to Getting Started and click Copy snippet to clipboard.
Next, replace this placeholder with your own code snippet:
# Replace with your code snippet
import validmind as vm
vm.init(
api_host="...",
api_key="...",
api_secret="...",
project="...",
)2024-04-10 17:28:10,149 - INFO(validmind.api_client): Connected to ValidMind. Project: [Int. Tests] Time Series - Initial Validation (cltnl8c7v001j1omgyzmjrzhj)In this notebook we will run a collection of test suites that are available in the ValidMind Developer Framework. Test suites group together a collection of tests that are relevant for a specific use case. In our case, we will run test different test suites for time series forecasting models. Once a test suite runs successfully, its results will be automatically uploaded to the ValidMind platform.
vm.test_suites.list_suites()| ID | Name | Description | Tests |
|---|---|---|---|
| classifier_model_diagnosis | ClassifierDiagnosis | Test suite for sklearn classifier model diagnosis tests | validmind.model_validation.sklearn.OverfitDiagnosis, validmind.model_validation.sklearn.WeakspotsDiagnosis, validmind.model_validation.sklearn.RobustnessDiagnosis |
| classifier_full_suite | ClassifierFullSuite | Full test suite for binary classification models. | validmind.data_validation.DatasetDescription, validmind.data_validation.DescriptiveStatistics, validmind.data_validation.PearsonCorrelationMatrix, validmind.data_validation.ClassImbalance, validmind.data_validation.Duplicates, validmind.data_validation.HighCardinality, validmind.data_validation.HighPearsonCorrelation, validmind.data_validation.MissingValues, validmind.data_validation.Skewness, validmind.data_validation.UniqueRows, validmind.data_validation.TooManyZeroValues, validmind.model_validation.ModelMetadata, validmind.data_validation.DatasetSplit, validmind.model_validation.sklearn.ConfusionMatrix, validmind.model_validation.sklearn.ClassifierPerformance, validmind.model_validation.sklearn.PermutationFeatureImportance, validmind.model_validation.sklearn.PrecisionRecallCurve, validmind.model_validation.sklearn.ROCCurve, validmind.model_validation.sklearn.PopulationStabilityIndex, validmind.model_validation.sklearn.SHAPGlobalImportance, validmind.model_validation.sklearn.MinimumAccuracy, validmind.model_validation.sklearn.MinimumF1Score, validmind.model_validation.sklearn.MinimumROCAUCScore, validmind.model_validation.sklearn.TrainingTestDegradation, validmind.model_validation.sklearn.ModelsPerformanceComparison, validmind.model_validation.sklearn.OverfitDiagnosis, validmind.model_validation.sklearn.WeakspotsDiagnosis, validmind.model_validation.sklearn.RobustnessDiagnosis |
| classifier_metrics | ClassifierMetrics | Test suite for sklearn classifier metrics | validmind.model_validation.ModelMetadata, validmind.data_validation.DatasetSplit, validmind.model_validation.sklearn.ConfusionMatrix, validmind.model_validation.sklearn.ClassifierPerformance, validmind.model_validation.sklearn.PermutationFeatureImportance, validmind.model_validation.sklearn.PrecisionRecallCurve, validmind.model_validation.sklearn.ROCCurve, validmind.model_validation.sklearn.PopulationStabilityIndex, validmind.model_validation.sklearn.SHAPGlobalImportance |
| classifier_model_validation | ClassifierModelValidation | Test suite for binary classification models. | validmind.model_validation.ModelMetadata, validmind.data_validation.DatasetSplit, validmind.model_validation.sklearn.ConfusionMatrix, validmind.model_validation.sklearn.ClassifierPerformance, validmind.model_validation.sklearn.PermutationFeatureImportance, validmind.model_validation.sklearn.PrecisionRecallCurve, validmind.model_validation.sklearn.ROCCurve, validmind.model_validation.sklearn.PopulationStabilityIndex, validmind.model_validation.sklearn.SHAPGlobalImportance, validmind.model_validation.sklearn.MinimumAccuracy, validmind.model_validation.sklearn.MinimumF1Score, validmind.model_validation.sklearn.MinimumROCAUCScore, validmind.model_validation.sklearn.TrainingTestDegradation, validmind.model_validation.sklearn.ModelsPerformanceComparison, validmind.model_validation.sklearn.OverfitDiagnosis, validmind.model_validation.sklearn.WeakspotsDiagnosis, validmind.model_validation.sklearn.RobustnessDiagnosis |
| classifier_validation | ClassifierPerformance | Test suite for sklearn classifier models | validmind.model_validation.sklearn.MinimumAccuracy, validmind.model_validation.sklearn.MinimumF1Score, validmind.model_validation.sklearn.MinimumROCAUCScore, validmind.model_validation.sklearn.TrainingTestDegradation, validmind.model_validation.sklearn.ModelsPerformanceComparison |
| cluster_full_suite | ClusterFullSuite | Full test suite for clustering models. | validmind.model_validation.ModelMetadata, validmind.data_validation.DatasetSplit, validmind.model_validation.sklearn.HomogeneityScore, validmind.model_validation.sklearn.CompletenessScore, validmind.model_validation.sklearn.VMeasure, validmind.model_validation.sklearn.AdjustedRandIndex, validmind.model_validation.sklearn.AdjustedMutualInformation, validmind.model_validation.sklearn.FowlkesMallowsScore, validmind.model_validation.sklearn.ClusterPerformanceMetrics, validmind.model_validation.sklearn.ClusterCosineSimilarity, validmind.model_validation.sklearn.SilhouettePlot, validmind.model_validation.ClusterSizeDistribution, validmind.model_validation.sklearn.HyperParametersTuning, validmind.model_validation.sklearn.KMeansClustersOptimization |
| cluster_metrics | ClusterMetrics | Test suite for sklearn clustering metrics | validmind.model_validation.ModelMetadata, validmind.data_validation.DatasetSplit, validmind.model_validation.sklearn.HomogeneityScore, validmind.model_validation.sklearn.CompletenessScore, validmind.model_validation.sklearn.VMeasure, validmind.model_validation.sklearn.AdjustedRandIndex, validmind.model_validation.sklearn.AdjustedMutualInformation, validmind.model_validation.sklearn.FowlkesMallowsScore, validmind.model_validation.sklearn.ClusterPerformanceMetrics, validmind.model_validation.sklearn.ClusterCosineSimilarity, validmind.model_validation.sklearn.SilhouettePlot |
| cluster_performance | ClusterPerformance | Test suite for sklearn cluster performance | validmind.model_validation.ClusterSizeDistribution |
| embeddings_full_suite | EmbeddingsFullSuite | Full test suite for embeddings models. | validmind.model_validation.ModelMetadata, validmind.data_validation.DatasetSplit, validmind.model_validation.embeddings.DescriptiveAnalytics, validmind.model_validation.embeddings.CosineSimilarityDistribution, validmind.model_validation.embeddings.ClusterDistribution, validmind.model_validation.embeddings.EmbeddingsVisualization2D, validmind.model_validation.embeddings.StabilityAnalysisRandomNoise, validmind.model_validation.embeddings.StabilityAnalysisSynonyms, validmind.model_validation.embeddings.StabilityAnalysisKeyword, validmind.model_validation.embeddings.StabilityAnalysisTranslation |
| embeddings_metrics | EmbeddingsMetrics | Test suite for embeddings metrics | validmind.model_validation.ModelMetadata, validmind.data_validation.DatasetSplit, validmind.model_validation.embeddings.DescriptiveAnalytics, validmind.model_validation.embeddings.CosineSimilarityDistribution, validmind.model_validation.embeddings.ClusterDistribution, validmind.model_validation.embeddings.EmbeddingsVisualization2D |
| embeddings_model_performance | EmbeddingsPerformance | Test suite for embeddings model performance | validmind.model_validation.embeddings.StabilityAnalysisRandomNoise, validmind.model_validation.embeddings.StabilityAnalysisSynonyms, validmind.model_validation.embeddings.StabilityAnalysisKeyword, validmind.model_validation.embeddings.StabilityAnalysisTranslation |
| hyper_parameters_optimization | KmeansParametersOptimization | Test suite for sklearn hyperparameters optimization | validmind.model_validation.sklearn.HyperParametersTuning, validmind.model_validation.sklearn.KMeansClustersOptimization |
| llm_classifier_full_suite | LLMClassifierFullSuite | Full test suite for LLM classification models. | validmind.data_validation.ClassImbalance, validmind.data_validation.Duplicates, validmind.data_validation.nlp.StopWords, validmind.data_validation.nlp.Punctuations, validmind.data_validation.nlp.CommonWords, validmind.data_validation.nlp.TextDescription, validmind.model_validation.ModelMetadata, validmind.data_validation.DatasetSplit, validmind.model_validation.sklearn.ConfusionMatrix, validmind.model_validation.sklearn.ClassifierPerformance, validmind.model_validation.sklearn.PermutationFeatureImportance, validmind.model_validation.sklearn.PrecisionRecallCurve, validmind.model_validation.sklearn.ROCCurve, validmind.model_validation.sklearn.PopulationStabilityIndex, validmind.model_validation.sklearn.SHAPGlobalImportance, validmind.model_validation.sklearn.MinimumAccuracy, validmind.model_validation.sklearn.MinimumF1Score, validmind.model_validation.sklearn.MinimumROCAUCScore, validmind.model_validation.sklearn.TrainingTestDegradation, validmind.model_validation.sklearn.ModelsPerformanceComparison, validmind.model_validation.sklearn.OverfitDiagnosis, validmind.model_validation.sklearn.WeakspotsDiagnosis, validmind.model_validation.sklearn.RobustnessDiagnosis, validmind.prompt_validation.Bias, validmind.prompt_validation.Clarity, validmind.prompt_validation.Conciseness, validmind.prompt_validation.Delimitation, validmind.prompt_validation.NegativeInstruction, validmind.prompt_validation.Robustness, validmind.prompt_validation.Specificity |
| prompt_validation | PromptValidation | Test suite for prompt validation | validmind.prompt_validation.Bias, validmind.prompt_validation.Clarity, validmind.prompt_validation.Conciseness, validmind.prompt_validation.Delimitation, validmind.prompt_validation.NegativeInstruction, validmind.prompt_validation.Robustness, validmind.prompt_validation.Specificity |
| nlp_classifier_full_suite | NLPClassifierFullSuite | Full test suite for NLP classification models. | validmind.data_validation.ClassImbalance, validmind.data_validation.Duplicates, validmind.data_validation.nlp.StopWords, validmind.data_validation.nlp.Punctuations, validmind.data_validation.nlp.CommonWords, validmind.data_validation.nlp.TextDescription, validmind.model_validation.ModelMetadata, validmind.data_validation.DatasetSplit, validmind.model_validation.sklearn.ConfusionMatrix, validmind.model_validation.sklearn.ClassifierPerformance, validmind.model_validation.sklearn.PermutationFeatureImportance, validmind.model_validation.sklearn.PrecisionRecallCurve, validmind.model_validation.sklearn.ROCCurve, validmind.model_validation.sklearn.PopulationStabilityIndex, validmind.model_validation.sklearn.SHAPGlobalImportance, validmind.model_validation.sklearn.MinimumAccuracy, validmind.model_validation.sklearn.MinimumF1Score, validmind.model_validation.sklearn.MinimumROCAUCScore, validmind.model_validation.sklearn.TrainingTestDegradation, validmind.model_validation.sklearn.ModelsPerformanceComparison, validmind.model_validation.sklearn.OverfitDiagnosis, validmind.model_validation.sklearn.WeakspotsDiagnosis, validmind.model_validation.sklearn.RobustnessDiagnosis |
| regression_metrics | RegressionMetrics | Test suite for performance metrics of regression metrics | validmind.data_validation.DatasetSplit, validmind.model_validation.ModelMetadata, validmind.model_validation.sklearn.PermutationFeatureImportance |
| regression_model_description | RegressionModelDescription | Test suite for performance metric of regression model of statsmodels library | validmind.data_validation.DatasetSplit, validmind.model_validation.ModelMetadata |
| regression_models_evaluation | RegressionModelsEvaluation | Test suite for metrics comparison of regression model of statsmodels library | validmind.model_validation.statsmodels.RegressionModelsCoeffs, validmind.model_validation.statsmodels.RegressionModelsPerformance |
| regression_models_comparison | RegressionModelsComparison | Test suite for regression models performance comparison | validmind.model_validation.sklearn.RegressionModelsPerformanceComparison |
| regression_full_suite | RegressionFullSuite | Full test suite for regression models. | validmind.data_validation.DatasetDescription, validmind.data_validation.DescriptiveStatistics, validmind.data_validation.PearsonCorrelationMatrix, validmind.data_validation.ClassImbalance, validmind.data_validation.Duplicates, validmind.data_validation.HighCardinality, validmind.data_validation.HighPearsonCorrelation, validmind.data_validation.MissingValues, validmind.data_validation.Skewness, validmind.data_validation.UniqueRows, validmind.data_validation.TooManyZeroValues, validmind.data_validation.DatasetSplit, validmind.model_validation.ModelMetadata, validmind.model_validation.sklearn.PermutationFeatureImportance, validmind.model_validation.sklearn.RegressionErrors, validmind.model_validation.sklearn.RegressionR2Square, validmind.model_validation.sklearn.RegressionModelsPerformanceComparison |
| regression_performance | RegressionPerformance | Test suite for regression model performance | validmind.model_validation.sklearn.RegressionErrors, validmind.model_validation.sklearn.RegressionR2Square |
| summarization_metrics | SummarizationMetrics | Test suite for Summarization metrics | validmind.model_validation.RougeMetrics, validmind.model_validation.TokenDisparity, validmind.model_validation.BleuScore, validmind.model_validation.BertScore, validmind.model_validation.ContextualRecall |
| tabular_dataset | TabularDataset | Test suite for tabular datasets. | validmind.data_validation.DatasetDescription, validmind.data_validation.DescriptiveStatistics, validmind.data_validation.PearsonCorrelationMatrix, validmind.data_validation.ClassImbalance, validmind.data_validation.Duplicates, validmind.data_validation.HighCardinality, validmind.data_validation.HighPearsonCorrelation, validmind.data_validation.MissingValues, validmind.data_validation.Skewness, validmind.data_validation.UniqueRows, validmind.data_validation.TooManyZeroValues |
| tabular_dataset_description | TabularDatasetDescription | Test suite to extract metadata and descriptive statistics from a tabular dataset | validmind.data_validation.DatasetDescription, validmind.data_validation.DescriptiveStatistics, validmind.data_validation.PearsonCorrelationMatrix |
| tabular_data_quality | TabularDataQuality | Test suite for data quality on tabular datasets | validmind.data_validation.ClassImbalance, validmind.data_validation.Duplicates, validmind.data_validation.HighCardinality, validmind.data_validation.HighPearsonCorrelation, validmind.data_validation.MissingValues, validmind.data_validation.Skewness, validmind.data_validation.UniqueRows, validmind.data_validation.TooManyZeroValues |
| text_data_quality | TextDataQuality | Test suite for data quality on text data | validmind.data_validation.ClassImbalance, validmind.data_validation.Duplicates, validmind.data_validation.nlp.StopWords, validmind.data_validation.nlp.Punctuations, validmind.data_validation.nlp.CommonWords, validmind.data_validation.nlp.TextDescription |
| time_series_data_quality | TimeSeriesDataQuality | Test suite for data quality on time series datasets | validmind.data_validation.TimeSeriesOutliers, validmind.data_validation.TimeSeriesMissingValues, validmind.data_validation.TimeSeriesFrequency |
| time_series_dataset | TimeSeriesDataset | Test suite for time series datasets. | validmind.data_validation.TimeSeriesOutliers, validmind.data_validation.TimeSeriesMissingValues, validmind.data_validation.TimeSeriesFrequency, validmind.data_validation.TimeSeriesLinePlot, validmind.data_validation.TimeSeriesHistogram, validmind.data_validation.ACFandPACFPlot, validmind.data_validation.SeasonalDecompose, validmind.data_validation.AutoSeasonality, validmind.data_validation.AutoStationarity, validmind.data_validation.RollingStatsPlot, validmind.data_validation.AutoAR, validmind.data_validation.AutoMA, validmind.data_validation.ScatterPlot, validmind.data_validation.LaggedCorrelationHeatmap, validmind.data_validation.EngleGrangerCoint, validmind.data_validation.SpreadPlot |
| time_series_model_validation | TimeSeriesModelValidation | Test suite for time series model validation. | validmind.data_validation.DatasetSplit, validmind.model_validation.ModelMetadata, validmind.model_validation.statsmodels.RegressionModelsCoeffs, validmind.model_validation.statsmodels.RegressionModelsPerformance, validmind.model_validation.statsmodels.RegressionModelForecastPlotLevels, validmind.model_validation.statsmodels.RegressionModelSensitivityPlot |
| time_series_multivariate | TimeSeriesMultivariate | This test suite provides a preliminary understanding of the features and relationship in multivariate dataset. It presents various multivariate visualizations that can help identify patterns, trends, and relationships between pairs of variables. The visualizations are designed to explore the relationships between multiple features simultaneously. They allow you to quickly identify any patterns or trends in the data, as well as any potential outliers or anomalies. The individual feature distribution can also be explored to provide insight into the range and frequency of values observed in the data. This multivariate analysis test suite aims to provide an overview of the data structure and guide further exploration and modeling. | validmind.data_validation.ScatterPlot, validmind.data_validation.LaggedCorrelationHeatmap, validmind.data_validation.EngleGrangerCoint, validmind.data_validation.SpreadPlot |
| time_series_sensitivity | TimeSeriesSensitivity | This test suite performs sensitivity analysis on a statsmodels OLS linear regression model by applying distinct shocks to each input variable individually and then computing the model's predictions. The aim of this test suite is to investigate the model's responsiveness to variations in its inputs. By juxtaposing the model's predictions under baseline and shocked conditions, users can visually evaluate the sensitivity of the model to changes in each variable. This kind of analysis can also shed light on potential model limitations, including over-reliance on specific variables or insufficient responsiveness to changes in inputs. As a result, this test suite can provide insights that may be beneficial for refining the model structure, improving its robustness, and ensuring a more reliable prediction performance. | validmind.model_validation.statsmodels.RegressionModelSensitivityPlot |
| time_series_univariate | TimeSeriesUnivariate | This test suite provides a preliminary understanding of the target variable(s) used in the time series dataset. It visualizations that present the raw time series data and a histogram of the target variable(s). The raw time series data provides a visual inspection of the target variable's behavior over time. This helps to identify any patterns or trends in the data, as well as any potential outliers or anomalies. The histogram of the target variable displays the distribution of values, providing insight into the range and frequency of values observed in the data. | validmind.data_validation.TimeSeriesLinePlot, validmind.data_validation.TimeSeriesHistogram, validmind.data_validation.ACFandPACFPlot, validmind.data_validation.SeasonalDecompose, validmind.data_validation.AutoSeasonality, validmind.data_validation.AutoStationarity, validmind.data_validation.RollingStatsPlot, validmind.data_validation.AutoAR, validmind.data_validation.AutoMA |
For our example use case we will run the following test suites:
time_series_datasettime_series_model_validationFederal Reserve Economic Data, or FRED, is a comprehensive database maintained by the Federal Reserve Bank of St. Louis. It offers a wide array of economic data from various sources, including U.S. government agencies and international organizations. The dataset encompasses numerous economic indicators across various categories such as employment, consumer price indices, money supply, and gross domestic product, among others.
FRED provides a valuable resource for researchers, policymakers, and anyone interested in understanding economic trends and conducting economic analysis. The platform also includes tools for data visualization, which can help users interpret complex economic data and identify trends over time.
The following code snippet imports a sample FRED dataset into a Pandas dataframe:
from validmind.datasets.regression import fred as demo_dataset
target_column = demo_dataset.target_column
feature_columns = demo_dataset.feature_columns
df = demo_dataset.load_data()
df.tail(10)| MORTGAGE30US | FEDFUNDS | GS10 | UNRATE | |
|---|---|---|---|---|
| DATE | ||||
| 2023-03-02 | 6.65 | NaN | NaN | NaN |
| 2023-03-09 | 6.73 | NaN | NaN | NaN |
| 2023-03-16 | 6.60 | NaN | NaN | NaN |
| 2023-03-23 | 6.42 | NaN | NaN | NaN |
| 2023-03-30 | 6.32 | NaN | NaN | NaN |
| 2023-04-01 | NaN | NaN | 3.46 | NaN |
| 2023-04-06 | 6.28 | NaN | NaN | NaN |
| 2023-04-13 | 6.27 | NaN | NaN | NaN |
| 2023-04-20 | 6.39 | NaN | NaN | NaN |
| 2023-04-27 | 6.43 | NaN | NaN | NaN |
Let’s see what tests are included on each test suite:
vm.test_suites.describe_suite("time_series_data_quality")| ID | Name | Description | Tests |
|---|---|---|---|
| time_series_data_quality | TimeSeriesDataQuality | Test suite for data quality on time series datasets | validmind.data_validation.TimeSeriesOutliers, validmind.data_validation.TimeSeriesMissingValues, validmind.data_validation.TimeSeriesFrequency |
vm.test_suites.describe_suite("time_series_univariate")| ID | Name | Description | Tests |
|---|---|---|---|
| time_series_univariate | TimeSeriesUnivariate | This test suite provides a preliminary understanding of the target variable(s) used in the time series dataset. It visualizations that present the raw time series data and a histogram of the target variable(s). The raw time series data provides a visual inspection of the target variable's behavior over time. This helps to identify any patterns or trends in the data, as well as any potential outliers or anomalies. The histogram of the target variable displays the distribution of values, providing insight into the range and frequency of values observed in the data. | validmind.data_validation.TimeSeriesLinePlot, validmind.data_validation.TimeSeriesHistogram, validmind.data_validation.ACFandPACFPlot, validmind.data_validation.SeasonalDecompose, validmind.data_validation.AutoSeasonality, validmind.data_validation.AutoStationarity, validmind.data_validation.RollingStatsPlot, validmind.data_validation.AutoAR, validmind.data_validation.AutoMA |
Use the ValidMind Developer Framework to initialize the dataset object:
vm_dataset = vm.init_dataset(
input_id="raw_dataset",
dataset=df,
target_column=demo_dataset.target_column,
)2024-04-10 17:28:10,243 - INFO(validmind.client): Pandas dataset detected. Initializing VM Dataset instance...Next, use the ValidMind Developer Framework to run the test suite for time series datasets:
config = {
# TIME SERIES DATA QUALITY PARAMS
"validmind.data_validation.TimeSeriesOutliers": {"params": {"zscore_threshold": 3}},
"validmind.data_validation.TimeSeriesMissingValues": {
"params": {"min_threshold": 2}
},
# TIME SERIES UNIVARIATE PARAMS
"validmind.data_validation.RollingStatsPlot": {"params": {"window_size": 12}},
"validmind.data_validation.SeasonalDecompose": {
"params": {"seasonal_model": "additive"}
},
"validmind.data_validation.AutoSeasonality": {
"params": {"min_period": 1, "max_period": 3}
},
"validmind.data_validation.AutoStationarity": {
"params": {"max_order": 3, "threshold": 0.05}
},
"validmind.data_validation.AutoAR": {"params": {"max_ar_order": 2}},
"validmind.data_validation.AutoMA": {"params": {"max_ma_order": 2}},
# TIME SERIES MULTIVARIATE PARAMS
"validmind.data_validation.LaggedCorrelationHeatmap": {
"params": {
"target_col": demo_dataset.target_column,
"independent_vars": demo_dataset.feature_columns,
}
},
"validmind.data_validation.EngleGrangerCoint": {"params": {"threshold": 0.05}},
}
full_suite = vm.run_test_suite(
"time_series_dataset",
inputs={
"dataset": vm_dataset,
},
config=config,
)No artists with labels found to put in legend. Note that artists whose label start with an underscore are ignored when legend() is called with no argument.No artists with labels found to put in legend. Note that artists whose label start with an underscore are ignored when legend() is called with no argument.No artists with labels found to put in legend. Note that artists whose label start with an underscore are ignored when legend() is called with no argument./Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/graphics/tsaplots.py:348: FutureWarning:
The default method 'yw' can produce PACF values outside of the [-1,1] interval. After 0.13, the default will change tounadjusted Yule-Walker ('ywm'). You can use this method now by setting method='ywm'.
/Users/andres/code/validmind-sdk/validmind/tests/data_validation/SeasonalDecompose.py:202: UserWarning:
No frequency could be inferred for variable 'MORTGAGE30US'. Skipping seasonal decomposition and plots for this variable.
2024-04-10 17:28:12,609 - INFO(validmind.tests.data_validation.SeasonalDecompose): Frequency of FEDFUNDS: MS2024-04-10 17:28:12,609 - INFO(validmind.tests.data_validation.SeasonalDecompose): Frequency of FEDFUNDS: MS2024-04-10 17:28:12,988 - INFO(validmind.tests.data_validation.SeasonalDecompose): Frequency of GS10: MS2024-04-10 17:28:12,988 - INFO(validmind.tests.data_validation.SeasonalDecompose): Frequency of GS10: MS2024-04-10 17:28:13,124 - INFO(validmind.tests.data_validation.SeasonalDecompose): Frequency of UNRATE: MS2024-04-10 17:28:13,124 - INFO(validmind.tests.data_validation.SeasonalDecompose): Frequency of UNRATE: MSNo artists with labels found to put in legend. Note that artists whose label start with an underscore are ignored when legend() is called with no argument.No artists with labels found to put in legend. Note that artists whose label start with an underscore are ignored when legend() is called with no argument.No artists with labels found to put in legend. Note that artists whose label start with an underscore are ignored when legend() is called with no argument.No artists with labels found to put in legend. Note that artists whose label start with an underscore are ignored when legend() is called with no argument.2024-04-10 17:28:13,634 - WARNING(validmind.tests.data_validation.AutoAR): Warning: MORTGAGE30US is not stationary. Results may be inaccurate.2024-04-10 17:28:13,634 - WARNING(validmind.tests.data_validation.AutoAR): Warning: MORTGAGE30US is not stationary. Results may be inaccurate./Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
A date index has been provided, but it has no associated frequency information and so will be ignored when e.g. forecasting.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
A date index has been provided, but it has no associated frequency information and so will be ignored when e.g. forecasting.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
A date index has been provided, but it has no associated frequency information and so will be ignored when e.g. forecasting.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
A date index has been provided, but it has no associated frequency information and so will be ignored when e.g. forecasting.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
2024-04-10 17:28:13,739 - WARNING(validmind.tests.data_validation.AutoAR): Warning: GS10 is not stationary. Results may be inaccurate.2024-04-10 17:28:13,739 - WARNING(validmind.tests.data_validation.AutoAR): Warning: GS10 is not stationary. Results may be inaccurate./Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
2024-04-10 17:28:13,972 - WARNING(validmind.tests.data_validation.AutoMA): Warning: MORTGAGE30US is not stationary. Results may be inaccurate.2024-04-10 17:28:13,972 - WARNING(validmind.tests.data_validation.AutoMA): Warning: MORTGAGE30US is not stationary. Results may be inaccurate./Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
A date index has been provided, but it has no associated frequency information and so will be ignored when e.g. forecasting.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
A date index has been provided, but it has no associated frequency information and so will be ignored when e.g. forecasting.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
A date index has been provided, but it has no associated frequency information and so will be ignored when e.g. forecasting.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
A date index has been provided, but it has no associated frequency information and so will be ignored when e.g. forecasting.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
A date index has been provided, but it has no associated frequency information and so will be ignored when e.g. forecasting.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
A date index has been provided, but it has no associated frequency information and so will be ignored when e.g. forecasting.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/statespace/sarimax.py:978: UserWarning:
Non-invertible starting MA parameters found. Using zeros as starting parameters.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
A date index has been provided, but it has no associated frequency information and so will be ignored when e.g. forecasting.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
A date index has been provided, but it has no associated frequency information and so will be ignored when e.g. forecasting.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
A date index has been provided, but it has no associated frequency information and so will be ignored when e.g. forecasting.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/statespace/sarimax.py:978: UserWarning:
Non-invertible starting MA parameters found. Using zeros as starting parameters.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
A date index has been provided, but it has no associated frequency information and so will be ignored when e.g. forecasting.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
A date index has been provided, but it has no associated frequency information and so will be ignored when e.g. forecasting.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
A date index has been provided, but it has no associated frequency information and so will be ignored when e.g. forecasting.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/statespace/sarimax.py:978: UserWarning:
Non-invertible starting MA parameters found. Using zeros as starting parameters.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/statespace/sarimax.py:978: UserWarning:
Non-invertible starting MA parameters found. Using zeros as starting parameters.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/statespace/sarimax.py:978: UserWarning:
Non-invertible starting MA parameters found. Using zeros as starting parameters.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/statespace/sarimax.py:978: UserWarning:
Non-invertible starting MA parameters found. Using zeros as starting parameters.
2024-04-10 17:28:15,882 - WARNING(validmind.tests.data_validation.AutoMA): Warning: GS10 is not stationary. Results may be inaccurate.2024-04-10 17:28:15,882 - WARNING(validmind.tests.data_validation.AutoMA): Warning: GS10 is not stationary. Results may be inaccurate./Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/statespace/sarimax.py:978: UserWarning:
Non-invertible starting MA parameters found. Using zeros as starting parameters.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/statespace/sarimax.py:978: UserWarning:
Non-invertible starting MA parameters found. Using zeros as starting parameters.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/statespace/sarimax.py:978: UserWarning:
Non-invertible starting MA parameters found. Using zeros as starting parameters.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/statespace/sarimax.py:978: UserWarning:
Non-invertible starting MA parameters found. Using zeros as starting parameters.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/statespace/sarimax.py:978: UserWarning:
Non-invertible starting MA parameters found. Using zeros as starting parameters.
# Sample frequencies to Monthly
resampled_df = df.resample("MS").last()
# Remove all missing values
nona_df = resampled_df.dropna()
# Take the first different across all variables
preprocessed_df = nona_df.diff().dropna()vm_dataset = vm.init_dataset(
input_id="preprocess_dataset",
dataset=preprocessed_df,
target_column=demo_dataset.target_column,
)
full_suite = vm.run_test_suite(
"time_series_dataset",
inputs={"dataset": vm_dataset},
config=config,
)2024-04-10 17:28:33,643 - INFO(validmind.client): Pandas dataset detected. Initializing VM Dataset instance...No artists with labels found to put in legend. Note that artists whose label start with an underscore are ignored when legend() is called with no argument.No artists with labels found to put in legend. Note that artists whose label start with an underscore are ignored when legend() is called with no argument.No artists with labels found to put in legend. Note that artists whose label start with an underscore are ignored when legend() is called with no argument./Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/graphics/tsaplots.py:348: FutureWarning:
The default method 'yw' can produce PACF values outside of the [-1,1] interval. After 0.13, the default will change tounadjusted Yule-Walker ('ywm'). You can use this method now by setting method='ywm'.
2024-04-10 17:28:34,844 - INFO(validmind.tests.data_validation.SeasonalDecompose): Frequency of MORTGAGE30US: MS2024-04-10 17:28:34,844 - INFO(validmind.tests.data_validation.SeasonalDecompose): Frequency of MORTGAGE30US: MS2024-04-10 17:28:35,127 - INFO(validmind.tests.data_validation.SeasonalDecompose): Frequency of FEDFUNDS: MS2024-04-10 17:28:35,127 - INFO(validmind.tests.data_validation.SeasonalDecompose): Frequency of FEDFUNDS: MS2024-04-10 17:28:35,350 - INFO(validmind.tests.data_validation.SeasonalDecompose): Frequency of GS10: MS2024-04-10 17:28:35,350 - INFO(validmind.tests.data_validation.SeasonalDecompose): Frequency of GS10: MS2024-04-10 17:28:35,483 - INFO(validmind.tests.data_validation.SeasonalDecompose): Frequency of UNRATE: MS2024-04-10 17:28:35,483 - INFO(validmind.tests.data_validation.SeasonalDecompose): Frequency of UNRATE: MSNo artists with labels found to put in legend. Note that artists whose label start with an underscore are ignored when legend() is called with no argument.No artists with labels found to put in legend. Note that artists whose label start with an underscore are ignored when legend() is called with no argument.No artists with labels found to put in legend. Note that artists whose label start with an underscore are ignored when legend() is called with no argument.No artists with labels found to put in legend. Note that artists whose label start with an underscore are ignored when legend() is called with no argument./Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:
No frequency information was provided, so inferred frequency MS will be used.
from validmind.datasets.regression import fred as demo_dataset
model_A, train_df_A, test_df_A = demo_dataset.load_model("fred_loan_rates_model_3")
model_B, train_df_B, test_df_B = demo_dataset.load_model("fred_loan_rates_model_4")# Initialize training and testing datasets for model A
vm_train_ds_A = vm.init_dataset(
input_id="train_a_dataset",
dataset=train_df_A,
target_column=demo_dataset.target_column,
)
vm_test_ds_A = vm.init_dataset(
input_id="test_a_dataset",
dataset=test_df_A,
target_column=demo_dataset.target_column,
)
# Initialize training and testing datasets for model B
vm_train_ds_B = vm.init_dataset(
input_id="train_b_dataset",
dataset=train_df_B,
target_column=demo_dataset.target_column,
)
vm_test_ds_B = vm.init_dataset(
input_id="test_b_dataset",
dataset=test_df_B,
target_column=demo_dataset.target_column,
)
# Initialize model A
vm_model_A = vm.init_model(
input_id="model_a",
model=model_A,
)
vm_train_ds_A.assign_predictions(model=vm_model_A)
vm_test_ds_A.assign_predictions(model=vm_model_A)
# Initialize model B
vm_model_B = vm.init_model(
input_id="model_b",
model=model_B,
)
vm_train_ds_B.assign_predictions(model=vm_model_B)
vm_test_ds_B.assign_predictions(model=vm_model_B)
models = [vm_model_A, vm_model_B]2024-04-10 17:28:54,767 - INFO(validmind.client): Pandas dataset detected. Initializing VM Dataset instance...2024-04-10 17:28:54,809 - INFO(validmind.client): Pandas dataset detected. Initializing VM Dataset instance...2024-04-10 17:28:54,847 - INFO(validmind.client): Pandas dataset detected. Initializing VM Dataset instance...2024-04-10 17:28:54,915 - INFO(validmind.client): Pandas dataset detected. Initializing VM Dataset instance...2024-04-10 17:28:55,744 - INFO(validmind.vm_models.dataset): Running predict()... This may take a while2024-04-10 17:28:55,745 - INFO(validmind.vm_models.dataset): Running predict()... This may take a while2024-04-10 17:28:55,782 - INFO(validmind.vm_models.dataset): Running predict()... This may take a while2024-04-10 17:28:55,782 - INFO(validmind.vm_models.dataset): Running predict()... This may take a whilevm.test_suites.describe_test_suite("time_series_model_validation")| ID | Name | Description | Tests |
|---|---|---|---|
| time_series_model_validation | TimeSeriesModelValidation | Test suite for time series model validation. | validmind.data_validation.DatasetSplit, validmind.model_validation.ModelMetadata, validmind.model_validation.statsmodels.RegressionModelsCoeffs, validmind.model_validation.statsmodels.RegressionModelsPerformance, validmind.model_validation.statsmodels.RegressionModelForecastPlotLevels, validmind.model_validation.statsmodels.RegressionModelSensitivityPlot |
config = {
"validmind.model_validation.statsmodels.RegressionModelForecastPlotLevels": {
"params": {
"transformation": "integrate",
}
},
"validmind.model_validation.statsmodels.RegressionModelSensitivityPlot": {
"params": {
"transformation": "integrate",
"shocks": [0.3],
}
},
"validmind.model_validation.statsmodels.RegressionModelsPerformance": {
"inputs": {
"in_sample_datasets": (vm_train_ds_A, vm_train_ds_B),
"out_of_sample_datasets": (vm_test_ds_A, vm_test_ds_B),
"models": models,
},
"params": {
"transformation": "integrate",
"shocks": [0.3],
},
},
"validmind.model_validation.statsmodels.RegressionModelForecastPlotLevels": {
"inputs": {
"datasets": (vm_train_ds_A, vm_test_ds_A),
"models": [vm_model_A],
},
},
"validmind.model_validation.statsmodels.RegressionModelSensitivityPlot": {
"inputs": {
"datasets": (vm_train_ds_A, vm_test_ds_A),
"models": [vm_model_A],
},
},
}
full_suite = vm.run_test_suite(
"time_series_model_validation",
inputs={
"dataset": vm_train_ds_A,
"datasets": (vm_train_ds_A, vm_test_ds_A),
"model": vm_model_A,
"models": models,
},
config=config,
)2024-04-10 17:28:55,901 - INFO(validmind.tests.model_validation.statsmodels.RegressionModelSensitivityPlot): {'transformation': None, 'shocks': [0.1]}2024-04-10 17:28:55,901 - INFO(validmind.tests.model_validation.statsmodels.RegressionModelSensitivityPlot): {'transformation': None, 'shocks': [0.1]}You can look at the results of this test suite right in the notebook where you ran the code, as you would expect. But there is a better way: view the prompt validation test results as part of your model documentation right in the ValidMind Platform UI:
In the Platform UI, go to the Documentation page for the model you registered earlier.
Expand 3. Model Development to review all test results.
What you can see now is a more easily consumable version of the prompt validation testing you just performed, along with other parts of your model documentation that still need to be completed.
If you want to learn more about where you are in the model documentation process, take a look at How do I use the framework?.